Conditional forced perspective in spherical video

ABSTRACT

A processor-implemented method in a digital media player includes receiving data defining a user&#39;s viewport in relation to spherical video frames displayed by the digital media player; and where a forced perspective event is to be triggered at the time of display of a spherical frame, triggering the forced perspective event conditionally based at least on the orientation of the user&#39;s viewport in relation to the spherical video frames. A digital media player includes processing structure for receiving data defining a user&#39;s viewport in relation to spherical video frames displayed by the digital media player; and processing structure for, where a forced perspective event is to be triggered at the time of display of a spherical frame, triggering the forced perspective event conditionally based at least on the orientation of the user&#39;s viewport in relation to the spherical video frames. A processor-readable medium embodying a computer program for a digital media player is provided. The computer program includes program code for receiving data defining a user&#39;s viewport in relation to spherical video frames displayed by the digital media player; and program code for, where a forced perspective event is to be triggered at the time of display of a spherical frame, triggering the forced perspective event conditionally based at least on the orientation of the user&#39;s viewport in relation to the spherical video frames.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.16/504,820 filed on Jul. 8, 2019, which claims priority under 35 U.S.C.120 to U.S. Provisional Patent Application Ser. No. 62/716,316 filed onAug. 8, 2018, the contents of which are incorporated by reference intheir entirety.

FIELD OF THE INVENTION

The following relates generally to electronic data processing, and moreparticularly to systems and methods for processing digital video.

BACKGROUND OF THE INVENTION

A wide variety of computing devices such as gaming consoles,virtual-reality equipment, augmented-reality equipment, mixed-realityequipment, smart televisions, set top boxes, desktop computers, laptops,smartphones, and specialty devices such as iPods, are available toconsumers. The computing capabilities of many of these devices can beharnessed by creative content producers to provide very rich, immersiveand interactive media content.

For example, filmmakers, digital content creators and technologydevelopers have been developing 360-video capture systems, correspondingauthoring tools and compatible media players to create and presentinteractive and immersive media experiences for a variety of platformsincluding virtual reality. Such video capture systems include multipleindividual but coordinated video cameras positioned in an array in whicheach camera has a unique position and field of view to collectivelycapture video that spans 360×180 degrees. Frames of the captured digitalvideo from the video cameras are synchronized and stitched togetherusing image processing algorithms to produce video frames each of whichcontains 360×180 content. Each of these video frames is typically storedin an equirectangular format, to facilitate straightforward projectiononto a geometry such as a spherical mesh for playback.

A user can be provided with the impression that he or she is positionedat the centre of the sphere looking outward towards the captured scenes,in a manner analogous to the position of the cameras during videocapture. In addition, the user may be provided with the ability toadjust his or her perspective and field of view—the viewport—such as byusing a mouse on a desktop-style system, a touch-display on a typicalsmartphone, or actual physical movement using virtual reality headgear(Head Mounted Display, or HMD), in order to face any part of the 360×180video that is being played back. In this way, the user can “look around”and in any direction will see the respective portions of the filmunfolding as it is played back just as one can look around in reality.

Processes for producing digital video from raw content such as thatcaptured by a 360-video capture system are well understood. Specialitysoftware tools are used to stitch together the content from thedifferent camera angles to produce the raw video. Then, the raw videocan be edited and spliced with other video, graphic overlays and thelike, on a computer workstation using software tools. When theauthor/editor is satisfied with the content, the digital video isconsidered “locked,” and post-production tools can be used to convertthe locked digital video into a form suitable for transmission, playbackand storage using various media players, devices and the like. Forexample, it is typical to encode raw video into a format such as MP4using H.264 or H.265 to compress the video so that the overall file inwhich it is contained is smaller and wieldy for storage andtransmission. Encoders are sets of hardware and software that receivethe original raw digital video content as input and that output anencoded digital video file. Transcoders are sets of hardware andsoftware that receive an encoded video file and re-encode the file intoa different encoded format. Decoders are sets of hardware and softwarethat receive an encoded video file, and extract each frame as pixel dataso that the pixel data can be inserted into a memory buffer which can belater stored in a frame buffer for subsequent display by a displaydevice. Together, coders/transcoders and decoders are typically referredto as codecs.

When authoring digital videos, an author/editor may wish to add certainevents which are to be triggered by a media player during playback.Parameters specifying such events and their exact timing may be storedas metadata in a file associated with the digital video, such as an XML(eXtensible Markup Language) or JSON (JavaScript Object Notation) file.Such event records may be identified according to frame number orplayback time; the events are to be triggered for execution by a mediaplayer at the time of display of a particular frame number or at aparticular time lapsed since beginning playback. Such frames may bereferred to as “event-triggering frames”. Various examples of suchevents and how they may be triggered are provided in United StatesPatent Application Publication No. 2018/0005447 to Wallner et al., andin United States Patent Application Publication No. 2018/0005449 toWallner et al., the contents of each of which are incorporated herein byreference in their entirety.

One example of an event that the author/editor may wish to trigger themedia player to conduct at particular times during the playback ofspherical video is a “forced perspective” event. Users' freedom to “lookaround” at different portions of the spherical video can competesomewhat with the author/editor's desire to properly present users withparticular visual narrative events or actions. That is, if users areexercising their freedom to look at any region of the spherical videoframes while an important narrative event or action is available to beseen at a different region within the spherical video frames, some usersmay miss seeing that narrative event or action. For preventing this, anauthor/editor during editing may add a forced perspective event inassociation with a particular frame, so that the media player whendisplaying that particular frame will automatically orient the user'sviewport at a particular juncture. This is used to force every user tothe perspective of the region of the scene that the author/editorintended. Once a forced perspective event is triggered, control over theviewport is typically immediately released back to the user, so the usercan continue to look around starting from the perspective to which theywere forced. In a sense, particularly within a scene, a user canperceive a forced perspective in spherical video as an effect similar toa “cut” in traditional flat video.

When, during playback and just prior to a forced perspective event, auser's existing perspective (as defined by his or her viewport) is quitedifferent from that to which the user will be forced, the subsequentautomatic forcing of that user's perspective is not typicallyexperienced as jarring. Users tend to accept such “cuts” as a matter ofcourse, as they do with traditional flat video. However, as withtraditional flat video, when a user's perspective is similar to that towhich the user will be forced, the subsequent forcing of that user'sperspective can be experienced as jarring. While such automatic cuts maybe intended by the author/editor as tools for imparting style oradvancing narrative (such as deliberately injecting jump cuts), it isdesirable that the author/editor maintain control over whether such cutsoccur. However, with spherical video, since the author/editor cannotanticipate with certainty in which direction every user who will everview the spherical digital video will be gazing, there is a risk thatsome users will experience unintended cuts, whereas other users willnot.

SUMMARY OF THE INVENTION

In accordance with an aspect, there is provided a processor-implementedmethod in a digital media player, the method comprising receiving datadefining a user's viewport in relation to spherical video framesdisplayed by the digital media player; and where a forced perspectiveevent is to be triggered at the time of display of a spherical frame,triggering the forced perspective event conditionally based at least onthe orientation of the user's viewport in relation to the sphericalvideo frames.

In an embodiment, triggering the forced perspective event conditionallycomprises triggering the forced perspective event only if the user'sviewport does not coincide with an exclusion zone associated with thespherical frame.

In an embodiment, triggering the forced perspective event conditionallycomprises triggering the forced perspective event only if the user'sviewport does not coincide for a threshold duration with an exclusionzone associated with the spherical frame.

In an embodiment, triggering the forced perspective event conditionallycomprises triggering the forced perspective event only if the user'sviewport coincides with an inclusion zone associated with the sphericalframe.

In an embodiment, the processor-implemented method further comprises,where one of a plurality of forced perspective events is to be triggeredat the time of display of the spherical frame, selecting none or one ofthe plurality of forced perspective events to be triggered conditionallybased at least on the orientation of the user's viewport in relation tothe spherical video frames, the selecting comprising selecting a forcedperspective event only if the user's viewport coincides with aninclusion zone associated with the selected forced perspective event;and in the event that one of the plurality of forced perspective eventsis selected, triggering the selected forced perspective event.

In an embodiment, the processor-implemented method further comprises,where one of a plurality of forced perspective events is to be triggeredat the time of display of the spherical frame, selecting none or one ofthe plurality of forced perspective events to be triggered conditionallybased at least on the orientation of the user's viewport in relation tothe spherical video frames, the selecting comprising selecting a forcedperspective event only if the user's viewport coincides for a thresholdduration with an inclusion zone associated with the selected forcedperspective event; and in the event that one of the plurality of forcedperspective events is selected, triggering the selected forcedperspective event.

According to another aspect, there is provided a digital media playercomprising processing structure for receiving data defining a user'sviewport in relation to spherical video frames displayed by the digitalmedia player; and processing structure for, where a forced perspectiveevent is to be triggered at the time of display of a spherical frame,triggering the forced perspective event conditionally based at least onthe orientation of the user's viewport in relation to the sphericalvideo frames.

In an embodiment, the processing structure for triggering the forcedperspective event conditionally comprises processing structure fortriggering the forced perspective event only if the user's viewport doesnot coincide with an exclusion zone associated with the spherical frame.

In an embodiment, the processing structure for triggering the forcedperspective event conditionally comprises processing structure fortriggering the forced perspective event only if the user's viewport doesnot coincide for a threshold duration with an exclusion zone associatedwith the spherical frame.

In an embodiment, the processing structure for triggering the forcedperspective event conditionally comprises processing structure fortriggering the forced perspective event only if the user's viewportcoincides with an inclusion zone associated with the spherical frame.

In an embodiment, the digital media player further comprises processingstructure for, where one of a plurality of forced perspective events isto be triggered at the time of display of the spherical frame, selectingnone or one of the plurality of forced perspective events to betriggered conditionally based at least on the orientation of the user'sviewport in relation to the spherical video frames, the selectingcomprising selecting a forced perspective event only if the user'sviewport coincides with an inclusion zone associated with the selectedforced perspective event; and processing structure for, in the eventthat one of the plurality of forced perspective events is selected,triggering the selected forced perspective event.

In an embodiment, the digital media player further comprises processingstructure for, where one of a plurality of forced perspective events isto be triggered at the time of display of the spherical frame, selectingnone or one of the plurality of forced perspective events to betriggered conditionally based at least on the orientation of the user'sviewport in relation to the spherical video frames, the selectingcomprising selecting a forced perspective event only if the user'sviewport coincides for a threshold duration with an inclusion zoneassociated with the selected forced perspective event; and processingstructure for, in the event that one of the plurality of forcedperspective events is selected, triggering the selected forcedperspective event.

According to another aspect, there is provided a processor-readablemedium embodying a computer program for a digital media player, thecomputer program comprising program code for receiving data defining auser's viewport in relation to spherical video frames displayed by thedigital media player; and program code for, where a forced perspectiveevent is to be triggered at the time of display of a spherical frame,triggering the forced perspective event conditionally based at least onthe orientation of the user's viewport in relation to the sphericalvideo frames.

In an embodiment, the program code for triggering the forced perspectiveevent conditionally comprises program code for triggering the forcedperspective event only if the user's viewport does not coincide with anexclusion zone associated with the spherical frame.

In an embodiment, the program code for triggering the forced perspectiveevent conditionally comprises program code for triggering the forcedperspective event only if the user's viewport does not coincide for athreshold duration with an exclusion zone associated with the sphericalframe.

In an embodiment, the program code for triggering the forced perspectiveevent conditionally comprises program code for triggering the forcedperspective event only if the user's viewport coincides with aninclusion zone associated with the spherical frame.

In an embodiment, the processor-readable medium further comprisesprogram code for, where one of a plurality of forced perspective eventsis to be triggered at the time of display of the spherical frame,selecting none or one of the plurality of forced perspective events tobe triggered conditionally based at least on the orientation of theuser's viewport in relation to the spherical video frames, the selectingcomprising selecting a forced perspective event only if the user'sviewport coincides with an inclusion zone associated with the selectedforced perspective event; and program code for, in the event that one ofthe plurality of forced perspective events is selected, triggering theselected forced perspective event.

In an embodiment, the processor-readable medium further comprisesprogram code for, where one of a plurality of forced perspective eventsis to be triggered at the time of display of the spherical frame,selecting none or one of the plurality of forced perspective events tobe triggered conditionally based at least on the orientation of theuser's viewport in relation to the spherical video frames, the selectingcomprising selecting a forced perspective event only if the user'sviewport coincides for a threshold duration with an inclusion zoneassociated with the selected forced perspective event; and program codefor, in the event that one of the plurality of forced perspective eventsis selected, triggering the selected forced perspective event.

Systems, methods and processor-readable media disclosed herein areapplicable to digital video content whether it is intended to bedownloaded in its entirety, streamed, and/or is live-streamed orbroadcast.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described with reference to theappended drawings in which:

FIG. 1 is a flowchart depicting steps in a process, according to anembodiment;

FIGS. 2A and 2B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example exclusion zone and an example user viewport both prior to andafter triggering of a forced perspective event;

FIGS. 3A and 3B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example exclusion zone and an example user viewport both prior to andafter triggering of a forced perspective event;

FIGS. 4A and 4B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example exclusion zone and an example user viewport when conditionsprevent triggering of a forced perspective event;

FIG. 5 is a screenshot of an authoring/editing screen where an exclusionzone for a forced perspective event is being defined in association witha frame as a patch on a sphere to which the spherical video frame willbe texture mapped;

FIGS. 6A and 6B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showinganother example exclusion zone and an example user viewport both priorto and after triggering of a forced perspective event;

FIGS. 7A and 7B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showinganother example exclusion zone and an example user viewport both priorto and after triggering of a forced perspective event;

FIGS. 8A and 8B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showinganother example exclusion zone and an example user viewport whenconditions prevent triggering of a forced perspective event;

FIG. 9 is a screenshot of an authoring/editing screen where an exclusionzone for a forced perspective event is being defined in association witha frame as a pane extending between points on a sphere to which thespherical video frame will be texture mapped;

FIG. 10 is a schematic diagram of a computing system according to anembodiment;

FIG. 11 is a flowchart depicting steps in a process, according to anembodiment;

FIGS. 12A and 12B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport when conditionsprevent triggering of a forced perspective event;

FIGS. 13A and 13B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport when conditionsprevent triggering of a forced perspective event;

FIGS. 14A and 14B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport both prior to andafter triggering of a forced perspective event;

FIG. 15 is a top plan view diagram of a sphere on which spherical videois texture mapped for display by a media player, showing multipleexample inclusion zones A, B, C and D, each associated with a respectiveforced perspective event;

FIGS. 16A and 16B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding withinclusion zone A thereby triggering a respective forced perspectiveevent associated with inclusion zone A;

FIGS. 17A and 17B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding withinclusion zone C thereby triggering a respective forced perspectiveevent associated with inclusion zone C;

FIGS. 18A and 18B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding with noneof inclusion zones A, B, C and D, thereby representing conditionspreventing triggering of any forced perspective events; and

FIG. 19 is a flowchart depicting steps in a process, according to anembodiment.

DETAILED DESCRIPTION

FIG. 1 is a flowchart depicting steps in a process 90 for processingdigital video for display on a digital media player, according to anembodiment. In this embodiment, during process 90, spherical frames ofdigital video are displayed (step 100) and user viewport data isreceived (step 200). The user viewport data is indicative of in whichdirection the user using the digital media player is gazing. A viewportmay be thought of as the region of the spherical video that a user cansee at any given moment. In this embodiment, the user viewport dataincludes a first angle within the range of 0 . . . 360 degrees (for aleft to right reading) and a second angle within the range of 0 . . .180 degrees (for an up to down reading). For example, a given reading ofuser viewport data may be (45, 90) indicating that, with respect to thespherical video frames, the centre of the user's viewport is at 45 of360 degrees (halfway between looking forward and looking rightward) andat 90 of 180 degrees (straight horizontal, as opposed to upwards ordownwards). A few seconds later, a given reading of user viewport datamight, for example, be (180, 45), indicating the user has changedhis/her orientation and is now looking rearward and somewhat upwards.

The frequency with which user viewport data is obtained will depend onthe equipment being used by a user to view the spherical video, theimplementing computing equipment for capturing the viewpoint data, andthe overall needs of the system, but may be, for example, 120 readingsper second. The angle of the centre of the user's orientation and theknown visual field of a user (the known extent of the viewport), as wellas any limitations to the viewport imposed by equipment itself such asscreens, establish the user's viewport. Generally speaking, the viewportsize is fixed during a playback sequence, whereas its angles arecontrolled by the user as the user “looks around”.

In the event that a forced perspective event is to be triggered (step300) upon display of a particular frame, it is then determined at step400 whether the viewport data coincides with an exclusion zone that isassociated with the forced perspective event.

In the event that the viewport data coincides with the exclusion zoneassociated with the forced perspective event (step 400), the process 90continues without executing the forced perspective event to displayspherical frames (step 100). Otherwise, the forced perspective event isexecuted (step 500). As such, the triggering of the forced perspectiveevent is conditional on whether the user's viewpoint does not coincidewith the exclusion zone.

During determining whether viewport data coincides with the exclusionzone, process 90 determines the extent of the exclusion zone associatedwith the forced perspective event, and then determines whether the angleof the user's viewport coincides with a location within the exclusionzone. This may be done by creating a ray from the user's location in themiddle of the sphere (0,0,0) along a trajectory passing from thislocation at angles corresponding to the user viewport data. In the eventthat the exclusion zone is defined as a patch (i.e. a region) of thesphere, viewport data coincides with the exclusion zone if the rayintersects the patch. In the event that the exclusion zone is defined asa pane extending between points on the sphere (or a pane simply definedanywhere between the user's location at the middle of the sphere and theouter boundary of the sphere, rather than the pane having necessarily tocontact the outer boundary of the sphere), viewport data coincides withthe exclusion zone if the ray intersects the pane. It will be understoodthat various ways of defining an exclusion zone, and for determiningwhether viewport data coincides with the exclusion zone, are possible.For example, in the event that the exclusion zone is defined as a paneas explained above, process 90 may first project the pane onto thesphere as a patch and determine at the time of the forced perspectiveevent whether the ray intersects that patch. As another example, in theevent that the exclusion zone is defined as a patch as explained above,process 90 may first translate the patch into a pane and determine atthe time of the forced perspective event whether the ray intersects thepane. Alternatives are possible.

It will be noted that a given forced perspective event may not beassociated with an exclusion zone. In such an event, there will be nocondition placed on the triggering of the forced perspective event thatrelates to the orientation of the user's viewport. Other conditions maybe placed on the triggering of the forced perspective event.

FIGS. 2A and 2B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, with anexample user viewport viewing angle not coinciding with an exclusionzone at the time of a forced perspective event such that the forcedperspective event is triggered, the exclusion zone being defined as apatch on the sphere. The forced perspective event is triggered becausethe angle of the viewport does not coincide with (extend into, intersectetc.) the exclusion zone.

In FIGS. 2A and 2B and similar figures set out below, the relative sizesof the exclusion zone and the user's viewport are not to scale and areso illustrated simply for ease of understanding. For example, anexclusion zone may be defined to be a small fraction of the size of aviewport, or an exclusion zone may be defined to be larger than the sizeof the viewport, depending on the requirements of the author/editor.Furthermore, while in this embodiment the angle to which perspective isto be forced corresponds to the middle of the exclusion zone,alternatives may be made available to an author/editor whereby the angleto which perspective is to be forced does not correspond to the middleof the exclusion zone.

FIGS. 3A and 3B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, withanother example user viewport viewing angle not coinciding with anexclusion zone at the time of a forced perspective event such that theforced perspective event is executed, the exclusion zone being definedas a patch on the sphere. While there is overlap of the exclusion zoneand the user's viewport, in this embodiment the viewport is notconsidered to coincide because the centre of the viewport does notcoincide with the exclusion zone. That is, the forced perspective eventis triggered because the angle of the viewport does not coincide with(extend into, intersect etc.) the exclusion zone.

FIGS. 4A and 4B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, withanother example user viewport viewing angle coinciding with an exclusionzone at the time of a forced perspective event such that the forcedperspective event is not triggered, the exclusion zone being defined asa patch on the sphere. The forced perspective event is not triggeredbecause the angle of the viewport coincides with the exclusion zone. Inthis case, a ray extending from the user at (0,0,0) along the viewportangle intersects the patch defining the exclusion zone.

FIG. 5 is a screenshot of an authoring/editing screen where an exclusionzone for a forced perspective event is being defined in association witha frame as a patch on a sphere to which the spherical video frame willbe texture mapped. In this embodiment, the author/editor can establishthe exclusion zone as a hit box for which “hits” execute no events,thereby overriding execution of a forced perspective event that is alsoassociated with this frame. More particularly, the hit box is associatedwith a particular frame, specified by degrees of orientation (H Rot, VRot) and a width and a height. The hit box (in this example, Hitbox 29)is associated with an In Frame 13694 and an Out frame 14450 (thus havinga duration of 756 frames), which provides that the hit box will, asframes 13694 to 14450 are being displayed, cause any forced perspectiveevents associated with the frames not to be executed should the user's(the eventual viewer of the video) gaze coincide with the hit box.

FIGS. 6A and 6B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, with anexample user viewport viewing angle not coinciding with an exclusionzone at the time of a forced perspective event such that the forcedperspective event is triggered, the exclusion zone being defined as apane. The forced perspective event is triggered because the angle of theviewport does not coincide with (extend into, intersect etc.) theexclusion zone.

FIGS. 7A and 7B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, withanother example user viewport viewing angle not coinciding with anexclusion zone at the time of a forced perspective event such that theforced perspective event is triggered, the exclusion zone being definedas a pane. While there is overlap of the exclusion zone and the user'sviewport, in this embodiment the viewport is not considered to coincidebecause the centre of the viewport does not coincide with the exclusionzone. That is, the forced perspective event is triggered because theangle of the viewport does not coincide with (extend into, intersectetc.) the exclusion zone.

FIGS. 8A and 8B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, withanother example user viewport viewing angle coinciding with an exclusionzone at the time of a forced perspective event such that the forcedperspective event is not triggered, the exclusion zone being defined asa pane. The forced perspective event is not triggered because the angleof the viewport coincides with the exclusion zone. In this case, a rayextending from the user at (0,0,0) along the viewport angle intersectsthe pane defining the exclusion zone.

FIG. 9 is a screenshot of an authoring/editing screen where an exclusionzone for a forced perspective event is being defined in association witha frame as a pane extending between points on a sphere to which thespherical video frame will be texture mapped. In this embodiment, theauthor/editor can establish the exclusion zone as a hit box for which“hits” execute no events, thereby overriding execution of a forcedperspective event that is also associated with this frame. Moreparticularly, the hit box is associated with a particular frame,specified by degrees of orientation (H Rot, V Rot) and a width and aheight. The hit box (in this example, Hitbox 29) is associated with anIn Frame 13694 and an Out frame 14450 (thus having a duration of 756frames), which provides that the hit box will, as frames 13694 to 14450are being displayed, cause any forced perspective events associated withthe frames not to be executed should the user's (the eventual viewer ofthe video) gaze coincide with the hit box.

In this embodiment, process 90 is executed on one or more systems suchas special purpose computing system 1000 shown in FIG. 2. Computingsystem 1000 may also be specially configured with software applicationsand hardware components to enable a user to play media such as digitalvideo, as well as to decode and/or transcode the digital video from andinto various formats such as MP4, AVI, MOV, WEBM and using a selectedcompression algorithm such as H.264 or H. 265 and according to variousselected parameters, thereby to view the digital video as desired for aparticular application, media player, or platform.

Computing system 1000 includes a bus 1010 or other communicationmechanism for communicating information, and a processor 1018 coupledwith the bus 1010 for processing the information. The computing system1000 also includes a main memory 1004, such as a random access memory(RAM) or other dynamic storage device (e.g., dynamic RAM (DRAM), staticRAM (SRAM), and synchronous DRAM (SDRAM)), coupled to the bus 1010 forstoring information and instructions to be executed by processor 1018.In addition, the main memory 1004 may be used for storing temporaryvariables or other intermediate information during the execution ofinstructions by the processor 1018. Processor 1018 may include memorystructures such as registers for storing such temporary variables orother intermediate information during execution of instructions. Thecomputing system 1000 further includes a read only memory (ROM) 1006 orother static storage device (e.g., programmable ROM (PROM), erasablePROM (EPROM), and electrically erasable PROM (EEPROM)) coupled to thebus 1010 for storing static information and instructions for theprocessor 1018.

The computing system 1000 also includes a disk controller 1008 coupledto the bus 1010 to control one or more storage devices for storinginformation and instructions, such as a magnetic hard disk 1022 and/or asolid state drive (SSD) and/or a flash drive, and a removable mediadrive 1024 (e.g., solid state drive such as USB key or external harddrive, floppy disk drive, read-only compact disc drive, read/writecompact disc drive, compact disc jukebox, tape drive, and removablemagneto-optical drive). The storage devices may be added to thecomputing system 1000 using an appropriate device interface (e.g.,Serial ATA (SATA), peripheral component interconnect (PCI), smallcomputing system interface (SCSI), integrated device electronics (IDE),enhanced-IDE (E-IDE), direct memory access (DMA), ultra-DMA, as well ascloud-based device interfaces).

The computing system 1000 may also include special purpose logic devices(e.g., application specific integrated circuits (ASICs)) or configurablelogic devices (e.g., simple programmable logic devices (SPLDs), complexprogrammable logic devices (CPLDs), and field programmable gate arrays(FPGAs)).

The computing system 1000 also includes a display controller 1002coupled to the bus 1010 to control a display 1012, such as an LED (lightemitting diode) screen, organic LED (OLED) screen, liquid crystaldisplay (LCD) screen or some other device suitable for displayinginformation to a computer user. In this embodiment, display controller1002 incorporates a dedicated graphics processing unit (GPU) forprocessing mainly graphics-intensive or other highly-paralleloperations. Such operations may include rendering by applying texturing,shading and the like to wireframe objects including polygons such asspheres and cubes thereby to relieve processor 1018 of having toundertake such intensive operations at the expense of overallperformance of computing system 1000. The GPU may incorporate dedicatedgraphics memory for storing data generated during its operations, andincludes a frame buffer RAM memory for storing processing results asbitmaps to be used to activate pixels of display 1012. The GPU may beinstructed to undertake various operations by applications running oncomputing system 1000 using a graphics-directed application programminginterface (API) such as OpenGL, Direct3D and the like.

The computing system 1000 includes input devices, such as a keyboard1014 and a pointing device 1016, for interacting with a computer userand providing information to the processor 1018. The pointing device1016, for example, may be a mouse, a trackball, or a pointing stick forcommunicating direction information and command selections to theprocessor 1018 and for controlling cursor movement on the display 1012.The computing system 1000 may employ a display device that is coupledwith an input device, such as a touch screen. Other input devices may beemployed, such as those that provide data to the computing system viawires or wirelessly, such as gesture detectors including infrareddetectors, gyroscopes, accelerometers, radar/sonar and the like. Aprinter may provide printed listings of data stored and/or generated bythe computing system 1000.

The computing system 1000 performs a portion or all of the processingsteps discussed herein in response to the processor 1018 and/or GPU ofdisplay controller 1002 executing one or more sequences of one or moreinstructions contained in a memory, such as the main memory 1004. Suchinstructions may be read into the main memory 1004 from anotherprocessor readable medium, such as a hard disk 1022 or a removable mediadrive 1024. One or more processors in a multi-processing arrangementsuch as computing system 1000 having both a central processing unit andone or more graphics processing unit may also be employed to execute thesequences of instructions contained in main memory 1004 or in dedicatedgraphics memory of the GPU. In alternative embodiments, hard-wiredcircuitry may be used in place of or in combination with softwareinstructions.

As stated above, the computing system 1000 includes at least oneprocessor readable medium or memory for holding instructions programmedaccording to the teachings of the invention and for containing datastructures, tables, records, or other data described herein. Examples ofprocessor readable media are solid state devices (SSD), flash-baseddrives, compact discs, hard disks, floppy disks, tape, magneto-opticaldisks, PROMs (EPROM, EEPROM, flash EPROM), DRAM, SRAM, SDRAM, or anyother magnetic medium, compact discs (e.g., CD-ROM), or any otheroptical medium, punch cards, paper tape, or other physical medium withpatterns of holes, a carrier wave (described below), or any other mediumfrom which a computer can read.

Stored on any one or on a combination of processor readable media,includes software for controlling the computing system 1000, for drivinga device or devices to perform the functions discussed herein, and forenabling the computing system 1000 to interact with a human user (e.g.,digital video author/editor). Such software may include, but is notlimited to, device drivers, operating systems, development tools, andapplications software. Such processor readable media further includesthe computer program product for performing all or a portion (ifprocessing is distributed) of the processing performed discussed herein.

The computer code devices of discussed herein may be any interpretableor executable code mechanism, including but not limited to scripts,interpretable programs, dynamic link libraries (DLLs), Java classes, andcomplete executable programs. Moreover, parts of the processing of thepresent invention may be distributed for better performance,reliability, and/or cost.

A processor readable medium providing instructions to a processor 1018may take many forms, including but not limited to, non-volatile media,volatile media, and transmission media. Non-volatile media includes, forexample, optical, magnetic disks, and magneto-optical disks, such as thehard disk 1022 or the removable media drive 1024. Volatile mediaincludes dynamic memory, such as the main memory 1004. Transmissionmedia includes coaxial cables, copper wire and fiber optics, includingthe wires that make up the bus 1010. Transmission media also may alsotake the form of acoustic or light waves, such as those generated duringradio wave and infrared data communications using various communicationsprotocols.

Various forms of processor readable media may be involved in carryingout one or more sequences of one or more instructions to processor 1018for execution. For example, the instructions may initially be carried ona magnetic disk of a remote computer. The remote computer can load theinstructions for implementing all or a portion of the present inventionremotely into a dynamic memory and send the instructions over a wired orwireless connection using a modem. A modem local to the computing system1000 may receive the data via wired Ethernet or wirelessly via Wi-Fi andplace the data on the bus 1010. The bus 1010 carries the data to themain memory 1004, from which the processor 1018 retrieves and executesthe instructions. The instructions received by the main memory 1004 mayoptionally be stored on storage device 1022 or 1024 either before orafter execution by processor 1018.

The computing system 1000 also includes a communication interface 1020coupled to the bus 1010. The communication interface 1020 provides atwo-way data communication coupling to a network link that is connectedto, for example, a local area network (LAN) 1500, or to anothercommunications network 2000 such as the Internet. For example, thecommunication interface 1020 may be a network interface card to attachto any packet switched LAN. As another example, the communicationinterface 1020 may be an asymmetrical digital subscriber line (ADSL)card, an integrated services digital network (ISDN) card or a modem toprovide a data communication connection to a corresponding type ofcommunications line. Wireless links may also be implemented. In any suchimplementation, the communication interface 1020 sends and receiveselectrical, electromagnetic or optical signals that carry digital datastreams representing various types of information.

The network link typically provides data communication through one ormore networks to other data devices, including without limitation toenable the flow of electronic information. For example, the network linkmay provide a connection to another computer through a local network1500 (e.g., a LAN) or through equipment operated by a service provider,which provides communication services through a communications network2000. The local network 1500 and the communications network 2000 use,for example, electrical, electromagnetic, or optical signals that carrydigital data streams, and the associated physical layer (e.g., CAT 5cable, coaxial cable, optical fiber, etc). The signals through thevarious networks and the signals on the network link and through thecommunication interface 1020, which carry the digital data to and fromthe computing system 1000, may be implemented in baseband signals, orcarrier wave based signals. The baseband signals convey the digital dataas unmodulated electrical pulses that are descriptive of a stream ofdigital data bits, where the term “bits” is to be construed broadly tomean symbol, where each symbol conveys at least one or more informationbits. The digital data may also be used to modulate a carrier wave, suchas with amplitude, phase and/or frequency shift keyed signals that arepropagated over a conductive media, or transmitted as electromagneticwaves through a propagation medium. Thus, the digital data may be sentas unmodulated baseband data through a “wired” communication channeland/or sent within a predetermined frequency band, different thanbaseband, by modulating a carrier wave. The computing system 1000 cantransmit and receive data, including program code, through thenetwork(s) 1500 and 2000, the network link and the communicationinterface 1020. Moreover, the network link may provide a connectionthrough a LAN 1500 to a mobile device 1300 such as a personal digitalassistant (PDA) laptop computer, or cellular telephone.

Computing system 1000 may be provisioned with or be in communicationwith live broadcast/streaming equipment that receives, in nearreal-time, a stream of digital video content captured in near real-timefrom a particular live event and associated with forced perspectiveevents and corresponding exclusion zones.

The electronic data store implemented in the database described hereinmay be one or more of a table, an array, a database, a structured datafile, an XML file, or some other functional data store, such as harddisk 1022 or removable media 1024.

A particular device appropriate for playback of a given digital videomay take any of a number of forms. For example, the media playercomputing system may process the digital video for playback using acentral processing unit (CPU) or both a CPU and a GPU, if appropriatelyequipped, or may be a hardware-based decoder. A media player computingsystem including a GPU would preferably support an abstractedapplication programming interface such as OpenGL for use by a mediaplayer application running on the computing system to instruct thegraphics processing unit of the media player computing system to conductvarious graphics-intensive or otherwise highly-parallel operations. Themedia player may take the form of a desktop or laptop computer, asmartphone, virtual reality headgear, or some other suitably provisionedand configured computing device.

Various forms of computing system could be employed to play back videocontent in particular, such as head mounted displays, augmented realitydevices, holographic displays, input/display devices that can interprethand and face gestures using machine vision as well as head movementsthrough various sensors, devices that can react to voice commands andthose that provide haptic feedback, surround sound audio and/or arewearables. Such devices may be capable of eye-tracking and of detectingand receiving neural signals that register brain waves, and/or otherbiometric signals as inputs that can be used to control visual and auralrepresentations of video content based on the selected viewport.

An XML or JSON metadata file containing events to be triggered by themedia player and other playback control data is made available fordownload in association with the encoded digital video file. In order toplay back the digital video as well as to trigger the events, the mediaplayer processes the digital video file thereby to reconstruct thecompressed frames of digital video and store the frames in video memoryof the media player for further processing.

Thereafter, the 360 video frames are texture-mapped by the media playerto the predetermined spherical mesh and the texture-mapped frame isinserted into the frame buffer RAM as a bitmap for display by thedisplay device of the media player.

Events, such as forced perspective events, associated with anevent-triggering frame are triggered by the media player as theevent-triggering frame is placed into the frame buffer. Elements such asgraphical overlays that are triggered to be rendered by certain eventsare rendered in real-time and in sync with the digital video frames withwhich the events are associated.

As for a computing device for displaying user interfaces (such as thoseshown in FIGS. 5 and 9) for permitting an author/editor to insert forcedperspective events and define exclusion zones as described above, acomputing device having various capabilities in addition to thosedescribed above in connection with computing device 1000 may beemployed. Such a computing system may similarly be specially configuredwith software applications and hardware components to enable a user toplay media such as digital video, as well as to decode and/or transcodethe digital video from and into various formats such as MP4, AVI, MOV,WEBM and using a selected compression algorithm such as H.264 or H. 265and according to various selected parameters, thereby to view thedigital video as desired for a particular application, media player, orplatform.

A user working on a particular composition—an author/editor—may insertvarious events including forced perspective events into the workflowusing a user interface of authoring/editing software tools, and maydefine an exclusion zone (or multiple exclusion zones) in connectionwith the forced perspective event that is associated with a particularframe. This may be done at any time, most usefully during editing inorder to preview the effect of events that are being associated withparticular frames. An exclusion zone can be as small or as large as anauthor/editor wishes, and once established in the user interface may bejoined with other parameters of the forced perspective event in terms ofcoordinates defining the bounds of the exclusion zone, and the type(such as patch or pane, as described above). In an embodiment, anexclusion zone can be a shape other than a rectangle or square, such asa circle or an oval. In an embodiment, upon an author/editor creating aforced perspective event, the software tool may automatically define anexclusion zone for the forced perspective event, having a default angle(centred, for example, on the frame with which the forced perspectiveevent is being associated), size and type (i.e., patch or pane) that theauthor/editor can then adjust or even delete as the author/editor seesfit. Such an automatic definition of the exclusion zone may be usefulfor author/editors who are new to the spherical medium, but suchautomatic definition may be a setting in the authoring/editingenvironment that can be toggled off by more experienced users who do notneed or want such default exclusion zones to be automatically defined atall.

The metadata file or a derivative of it is meant to accompany thedigital video file when downloaded or streamed to a media player forplayback, or may be located on the platform hardware hosting the mediaplayer. When accompanying the video file it could be included as part ofa header of a video file. However, this approach would requirere-rendering the video file in order to make modifications to themetadata and, where additional assets were required duringevent-triggering, such assets would have also to be tightly integratedin some way. Alternatively, when accompanying the video file themetadata file could simply have the same filename and path as the videofile, with a different file extension, such that the media player couldeasily find and handle the two files in cooperation with each other. Inthis embodiment, the metadata file is downloaded to the media player,parsed and represented in system memory as one or more events, includingone or more forced perspective events with respective exclusion zones,associated with a frame identifier that is/are to be triggered upondisplay of the decoded frame from which the corresponding frameidentifier has been parsed. Such a frame may be referred to as anevent-triggering frame, and there may be many such event-triggeringframes corresponding to one or more respective events to be executed bythe media player.

Although embodiments have been described with reference to the drawings,those of skill in the art will appreciate that variations andmodifications may be made without departing from the spirit, scope andpurpose of the invention as defined by the appended claims.

While, in embodiments described herein, a viewport was not considered tocoincide with an exclusion zone in the event that its centre did notcoincide with the exclusion zone, alternatives are possible. Forexample, implementations may register a viewport coinciding with anexclusion zone in the event that there was any overlap, a particularamount of overlap, or some other criterion. This may depend on therequirements of the author/editor, the relative sizes of the viewportand the exclusion zone, and other factors.

The principles described above are applicable to modifying whether aforced perspective event is triggered based on a condition related to inwhich direction a user is gazing at the time the forced perspectiveevent is to be triggered. While the concepts relating to an exclusionzone or zones have been described above, it is similarly possible todefine an inclusion zone or zones. An inclusion zone is similar to anexclusion zone, except that the associated forced perspective event isexecuted to orient the user's viewport to anywhere in the sphere theauthor/editor wishes only if the user is gazing at one of potentiallymultiple inclusion zones at the relevant time, and is otherwise notexecuted. In embodiments described above, a single inclusion zone iseffectively defined as the remainder of the spherical video frames notcoinciding with the defined exclusion zone patch or pane. However, inalternative embodiments each inclusion zone could be specificallydefined as a respective patch or pane such that the remainder of thespherical video frames area not covered by an inclusion zone wouldoperate, effectively, as an exclusion zone. In such a case, determiningwhether a user's viewport coincides with the exclusion zone couldinvolve determining whether the ray extending from the user's position(0,0,0) at the viewport angle would miss all such patches or panes.Other methods of automatically making this determination may beemployed.

FIG. 11 is a flowchart depicting steps in a process 90A for processingdigital video for display on a digital media player, according to analternative embodiment. Process 90A is similar to process 90 describedabove, with the exception that an inclusion zone is defined inassociation with a forced perspective event such that process 90Atriggers the forced perspective event only if the user's viewportcoincides with the inclusion zone. As such, in this embodiment, duringprocess 90A, spherical frames of digital video are displayed (step 100)and user viewport data is received (step 200). In the event that aforced perspective event is to be triggered (step 300) upon display of aparticular frame, it is then determined at step 400A whether theviewport data coincides with an inclusion zone that is associated withthe forced perspective event. In the event that the viewport data doesnot coincide with the inclusion zone associated with the forcedperspective event (step 400A), the process 90A continues withoutexecuting the forced perspective event to display spherical frames (step100). Otherwise, the forced perspective event is executed (step 500). Assuch, the triggering of the forced perspective event is conditional onwhether the user's viewpoint coincides with the inclusion zone.

FIGS. 12A and 12B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport when conditionsprevent triggering of a forced perspective event. In this case, theviewport viewing angle does not coincide with the inclusion zone at thetime of a forced perspective event such that the forced perspectiveevent is not triggered. The forced perspective event is not triggeredbecause the angle of the viewport does not coincide with (extend into,intersect etc.) the inclusion zone.

In FIGS. 12A and 12B and similar figures set out below, the relativesizes of the inclusion zone(s) and the user's viewport are not to scaleand are so illustrated simply for ease of understanding. For example, aninclusion zone may be defined to be a small fraction of the size of aviewport, or an inclusion zone may be defined to be larger than the sizeof the viewport, depending on the requirements of the author/editor.

FIGS. 13A and 13B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport when conditionsprevent triggering of a forced perspective event. In this case, theviewport viewing angle does not coincide with the inclusion zone at thetime of a forced perspective event such that the forced perspectiveevent is not triggered. The forced perspective event is not triggeredbecause the angle of the viewport does not coincide with (extend into,intersect etc.) the inclusion zone.

FIGS. 14A and 14B are top plan view diagrams of a sphere on whichspherical video is texture mapped for display by a media player, showingan example inclusion zone and an example user viewport both prior to andafter triggering of a forced perspective event. It can be seen that theangle to which perspective is to be forced does not correspond to theinclusion zone, in this example. An author/editor may wish for the angleto which perspective is to be forced to correspond in some way to theinclusion zone in order to induce a jump cut, for example, but may usethis mechanism in order to direct the user's attention when gazing in aparticular direction (the inclusion zone) to a particular otherdirection (some particular direction away from the inclusion zone).

FIG. 15 is a top plan view diagram of a sphere on which spherical videois texture mapped for display by a media player, showing multipleexample inclusion zones A, B, C and D, each associated with a respectiveforced perspective event. This arrangement provides for a “set” ofexclusion zones established by the author/editor in order to triggerdifferent forced perspective events based on in which direction the useris gazing at the time the associated frame (or sequence of frames) is tobe displayed. As such, a given frame (or sequence of frames) can beassociated by the author/editor with multiple forced perspective events,each of which are associated with a respective different inclusion zone.For example, if a user watching a scene has chosen to orient his or herviewport to coincide with inclusion zone A in order to watch character Awhile character A engages in an argument with character B, then a forcedperspective event associated with a particular frame showing theargument and with inclusion zone A can be triggered at the given time toautomatically orient the user's viewport to face a region of thespherical video containing character B. Otherwise, if the user'sviewport is not oriented to coincide with inclusion zone A, the forcedperspective event associated with inclusion zone A will not betriggered. Rather, either no forced perspective events will be triggereddue to the user's viewport coinciding with none of the inclusion zones,or some other forced perspective event will be triggered if the user'sviewport is oriented to coincide with any of the other inclusion zonesB, C or D.

In order to provide interesting and varied experiences while drawingattention to certain content, an author/editor may establish severalforced perspective events distributed across a given scene. For example,if the user's viewport was oriented to face character A, then the nextforced perspective event would cause the user's viewport to be orientedto face character B with whom character A is arguing. As the argument inthe scene progresses, the user may be compelled to keep his or herviewport fixed on character B such that a subsequent forced perspectiveevent with an inclusion zone that coincides with character B would causethe user's viewport to switch back to an orientation that is fixed oncharacter A. By contrast, a different user watching the same footagemight, at the beginning of the scene, have caused their viewport to beoriented to face character B. The next forced perspective event could beestablished by the author/editor in association with an inclusion zonecoinciding with character B that would in fact cause the orientation ofthe different user's viewport to switch to coincide with character A. Assuch, the author/editor will have established a sophisticated experiencewhereby two (or more) different users watching the same footage couldhave their perspectives forced differently thereby to receive slightlydifferent scene-length experiences based on where their respectiveviewports were originally oriented when the first forced perspectiveevent was to be triggered. The author/editor may use this functionalityto enable the experiences of two different users to diverge based onwhat each were originally looking at, and then perhaps to converge againas the author/editor sees fit such that all users are looking at thesame thing at the close of the scene, for example. This could be used invarious ways. For example, an author/editor may anticipate that a femaleuser may be inclined to gaze at attractive male characters in thefootage, and the author/editor could use this consideration inconjunction with inclusion zones and forced perspectives to try to guidethe female user's gaze to particular features in the footage as itprogresses. At the same time, the author/editor may anticipate that amale user may be inclined to gaze at attractive female characters in thefootage, such that the author/editor could use this consideration inconjunction with other inclusion zones and forced perspectives to try toguide the male user's gaze to particular other features in the footageas it progresses. Different users could therefore be guided alongdifferent experiences from the same footage, and would somewhat“self-select” which experience they would have based on their initialviewport orientation at the time the author/editor intended to begin the“guided” experience. As would be understood, such self-selecting wouldtypically, particularly during a first watching of the footage, not bedeliberate on the part of the users. However, later, two different usersmight with great interest discuss their respective experiences watchingthe same footage, having experienced it in contrasting ways, leadingperhaps to the two different users being compelled to watch the footageagain but to deliberately orient their viewports in a different way forthe second and any subsequent viewings thereby to be “taken along” asothers had been.

For example, one user originally fixated on an exotic car in somefootage may be automatically, over time, forced to perspectives betweena host in the scene discussing the car and back again to the car, overthe course of a scene. Another user originally fixated on airplanesflying overhead in the same footage may not be automatically forced toany different perspective. In this case, the author/editor may feel thata user who is not demonstrating particular interest in the exotic car atthe moment is more likely to enjoy continuing to gaze into the sky atthe airplanes and would rather not be wrenched away from thatexperience. Yet another user watching the same footage may be gazing atpedestrians walking along a sidewalk looking excitedly at the exotic carand, because of the orientation of this other user's viewport, thatuser's viewport may automatically be forced back and forth toperspectives of pedestrians on one side of the street and on the other,rather than between the exotic car itself and the host who is beingheard discussing it. At any point during such a scene if a user who hasnot previously been paying any attention to the exotic car decides togaze at it for a period of time, then the subsequent forced perspectivesthat are triggered in connection with an inclusion zone corresponding tothe car may allow the user to be “taken along” the above-mentionedseries of intermittent forced perspectives that have been arranged bythe author/editor with relevant inclusion zones to toggle between thatexotic car and the talkative host.

In embodiments, a particular forced perspective event associated with aninclusion zone may be triggered only if the user's viewport hascoincided with the inclusion zone for a threshold period of time. Thus,in such an embodiment, if a user is panning from left to right in thefootage and happens during this activity to cause his or her viewport totemporarily (i.e., less than the threshold period) coincide with aninclusion zone at the time of an associated forced perspective, thedigital media player will not trigger the forced perspective event. Onlyif the user's viewport coincides with the inclusion zone for at leastthe threshold period before the forced perspective event is to occurwill the user's viewport be automatically forced to the associatedperspective. This would, for example, enable an author/editor to setparameters on a forced perspective event to allow the eventual user to“look around” with a fair bit of freedom, but to be drawn into anarrative moment should the user provide enough indication (time spentlooking at something particular) that he or she might be interested inpaying attention to some feature of the footage that happens tocorrespond to an inclusion zone. Such a time threshold may be associatedby authors/editors with some forced perspective events and not withothers, and the amounts of time thresholds may be established by theauthor/editor individually for each forced perspective event andinclusion zone, or may be globally established for an entire scene orvideo.

It will be understood that the system for use by the author/editorduring editing will function to enable the author/editor to retain theflexibility to define inclusion zones and their forced perspectiveevents such that two or more inclusion zones may be associated withforced perspective events that orient the user's viewport in the samedirection. For example, an author/editor may wish to arrange the forcedperspective events and the inclusion zones such that a user watchingeither character A or character B during an argument would have theirperspective forced to face character C, who may be watching characters Aand B argue.

FIGS. 16A and 16B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding withinclusion zone A thereby triggering a respective forced perspectiveevent associated with inclusion zone A.

It will be noted that, in this example, the forced perspective eventassociated with inclusion zone A automatically causes the user'sviewport to be oriented in a direction that actually coincides withinclusion zone C. In this embodiment, inclusion zones may each beassociated by an author/editor with a sequence of frames, rather thanjust one frame. With the user's viewport having been automaticallyoriented to coincide with inclusion zone C, it would not generally bedesirable to, at the subsequent frame, immediately trigger the forcedperspective that is associated with inclusion zone C thereby toautomatically and immediately re-orient the user's viewport again. Thisis because the user's viewport will not have had a chance to “settle” inone orientation for long enough to allow the user to register what he orshe is being oriented to see before being forced into another viewportorientation. As such, the digital media player is configured todistinguish between orientations of the user's viewport that arecontrolled by the digital media player (i.e. that were forced due totriggering of a forced perspective event), and orientations of theuser's viewport that were voluntary (i.e. that were not forced due to atriggering of some forced perspective event). The digital media playercan therefore avoid an unintended automatic loop or cascade of forcedperspective events that would jar the user or thwart the author/editor'sintentions as to what the user's viewport should be oriented to settleon. This may be done by the digital media player being instructed orbeing configured to limit the number of triggered forced perspectiveevents to one per sequence of frames (such as one per 100 frames which,at 30 frames per second would be approximately one every 3 seconds),and/or to one per time period (such as one every approximately 3seconds), and/or to one per “set” of alternate forced perspectiveevents. Such thresholds may be established by the author/editor duringediting, or may be established as defaults.

FIGS. 17A and 17B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding withinclusion zone C thereby triggering a respective forced perspectiveevent associated with inclusion zone C.

FIGS. 18A and 18B are top plan view diagrams of the sphere and inclusionzones of FIG. 15, showing an example user viewport coinciding with noneof inclusion zones A, B, C and D, thereby representing conditionspreventing triggering of any forced perspective events.

FIG. 19 is a flowchart depicting steps in a process 90B for processingdigital video for display on a digital media player, according to analternative embodiment. Process 90B is similar to process 90A describedabove, with the exception that multiple forced perspective events A, B,C and D are defined in association with respective different inclusionzones A, B, C, D such that process 90B triggers one of the forcedperspective (FP) events A, B, C, D during display of a particular frame(or frame sequence) only if the user's viewport coincides with arespective inclusion zone A, B, C, D. Furthermore, a time parameter isdefined in association with a respective one of the forced perspectiveevents A, B, C, D—namely Time A, Time B, Time C and Time D. The timeparameter relates to a duration for which the user's viewport coincideswith a respective inclusion zone such that, only if (at the time oftriggering the forced perspective) the user's viewport has coincidedwith the inclusion zone for a duration that is at least as long as itsrespective time parameter (such as, for example, 2 seconds, or 60frames, or some other suitable amount to be established by theauthor/editor or by default) will the respective forced perspectiveevent in fact be triggered. During process 90B, spherical frames ofdigital video are displayed (step 100) and user viewport data isreceived (step 200). In the event that a forced perspective event is tobe triggered (step 300) upon display of a particular frame, it is thendetermined at step 400B whether the viewport data coincides with any ofinclusion zones A, B, C, D that have been defined in association withrespective forced perspective events A, B, C, D for that frame (or framesequence). In the event that the viewport data does not coincide withany such inclusion zones (step 400B), the process 90B continues withoutexecuting any forced perspective events to display spherical frames(step 100). Otherwise, process 90B continues to one of steps 410, 420,430, 440 to determine whether the user's viewport has coincided with therespective inclusion zone for the respective threshold time. In theevent that the viewport data has not coincided with the respectiveinclusion zone for the respective threshold time, the process 90Bcontinues without executing any forced perspective events to displayspherical frames (step 100). Otherwise, the respective forcedperspective event (one of FP Event A, FP Event B, FP Event C, FP EventD) is executed (i.e. triggered), as shown in one of steps 510, 520, 530,and 540.

It will be understood that alternative processes to process 90B arepossible. For example, a given inclusion zone may not have any timethreshold associated with it, while another inclusion zone does have atime threshold associated with it. Or, the process 90B could be executedwithout steps 410, 420, 430, 440 simply to execute respective forcedperspective events with no condition other than the condition that thatthe user's viewport coincides with the respective inclusion zone at thetime the forced perspective event is to be triggered.

It will be understood that the threshold period of time parameterreferred to above may be employed in connection with exclusion zonesalso, such that a forced perspective event associated with an exclusionzone would be executed even if the user happened to be gazing into anassociated exclusion zone, provided the user had been gazing into thatexclusion zone for less than a respective threshold period of timebeforehand. While the uses of a threshold time period for an exclusionzone may have fewer applications than for an inclusion zone, theflexibility could be useful for the author/editor. For example, a userquickly scanning his or her viewport through a particular exclusion zoneon his or her way to gaze at a different region may not tend toexperience a forced perspective event in the same way that a user whohas spent a threshold amount of time gazing through a viewport thatcoincides with the exclusion zone. One of these experiences could bemore or less jarring than the other. In this way, the author/editorcould fine-tune the digital video experience to account for differentusers' behaviours in this respect at the time during which the forcedperspective event is to be triggered.

The principles described herein are applicable to modifying whether aforced perspective event is triggered based on a condition related to inwhich direction a user is gazing at the time the forced perspectiveevent is to be triggered, and optionally the duration for which the userwas gazing in the direction.

What is claimed is:
 1. A processor-implemented method in a digital mediaplayer, the method comprising: receiving data defining a user's viewportin relation to spherical video frames displayed by the digital mediaplayer; and where a forced perspective event is to be triggered at thetime of display of a spherical frame, triggering the forced perspectiveevent conditionally based at least on the orientation of the user'sviewport in relation to the spherical video frames.
 2. Theprocessor-implemented method of claim 1, wherein triggering the forcedperspective event conditionally comprises triggering the forcedperspective event only if the user's viewport does not coincide with anexclusion zone associated with the spherical frame.
 3. Theprocessor-implemented method of claim 1, wherein triggering the forcedperspective event conditionally comprises triggering the forcedperspective event only if the user's viewport does not coincide for athreshold duration with an exclusion zone associated with the sphericalframe.
 4. The processor-implemented method of claim 1, whereintriggering the forced perspective event conditionally comprisestriggering the forced perspective event only if the user's viewportcoincides with an inclusion zone associated with the spherical frame. 5.The processor-implemented method of claim 1, further comprising: whereone of a plurality of forced perspective events is to be triggered atthe time of display of the spherical frame, selecting none or one of theplurality of forced perspective events to be triggered conditionallybased at least on the orientation of the user's viewport in relation tothe spherical video frames, the selecting comprising selecting a forcedperspective event only if the user's viewport coincides with aninclusion zone associated with the selected forced perspective event;and in the event that one of the plurality of forced perspective eventsis selected, triggering the selected forced perspective event.
 6. Theprocessor-implemented method of claim 1, further comprising: where oneof a plurality of forced perspective events is to be triggered at thetime of display of the spherical frame, selecting none or one of theplurality of forced perspective events to be triggered conditionallybased at least on the orientation of the user's viewport in relation tothe spherical video frames, the selecting comprising selecting a forcedperspective event only if the user's viewport coincides for a thresholdduration with an inclusion zone associated with the selected forcedperspective event; and in the event that one of the plurality of forcedperspective events is selected, triggering the selected forcedperspective event.
 7. A digital media player comprising: processingstructure for receiving data defining a user's viewport in relation tospherical video frames displayed by the digital media player; andprocessing structure for, where a forced perspective event is to betriggered at the time of display of a spherical frame, triggering theforced perspective event conditionally based at least on the orientationof the user's viewport in relation to the spherical video frames.
 8. Thedigital media player of claim 7, wherein the processing structure fortriggering the forced perspective event conditionally comprisesprocessing structure for triggering the forced perspective event only ifthe user's viewport does not coincide with an exclusion zone associatedwith the spherical frame.
 9. The digital media player of claim 7,wherein the processing structure for triggering the forced perspectiveevent conditionally comprises processing structure for triggering theforced perspective event only if the user's viewport does not coincidefor a threshold duration with an exclusion zone associated with thespherical frame.
 10. The digital media player of claim 7, wherein theprocessing structure for triggering the forced perspective eventconditionally comprises processing structure for triggering the forcedperspective event only if the user's viewport coincides with aninclusion zone associated with the spherical frame.
 11. The digitalmedia player of claim 7, further comprising: processing structure for,where one of a plurality of forced perspective events is to be triggeredat the time of display of the spherical frame, selecting none or one ofthe plurality of forced perspective events to be triggered conditionallybased at least on the orientation of the user's viewport in relation tothe spherical video frames, the selecting comprising selecting a forcedperspective event only if the user's viewport coincides with aninclusion zone associated with the selected forced perspective event;and processing structure for, in the event that one of the plurality offorced perspective events is selected, triggering the selected forcedperspective event.
 12. The digital media player of claim 7, furthercomprising: processing structure for, where one of a plurality of forcedperspective events is to be triggered at the time of display of thespherical frame, selecting none or one of the plurality of forcedperspective events to be triggered conditionally based at least on theorientation of the user's viewport in relation to the spherical videoframes, the selecting comprising selecting a forced perspective eventonly if the user's viewport coincides for a threshold duration with aninclusion zone associated with the selected forced perspective event;and processing structure for, in the event that one of the plurality offorced perspective events is selected, triggering the selected forcedperspective event.
 13. A processor-readable medium embodying a computerprogram for a digital media player, the computer program comprising:program code for receiving data defining a user's viewport in relationto spherical video frames displayed by the digital media player; andprogram code for, where a forced perspective event is to be triggered atthe time of display of a spherical frame, triggering the forcedperspective event conditionally based at least on the orientation of theuser's viewport in relation to the spherical video frames.
 14. Theprocessor-readable medium of claim 13, wherein the program code fortriggering the forced perspective event conditionally comprises programcode for triggering the forced perspective event only if the user'sviewport does not coincide with an exclusion zone associated with thespherical frame.
 15. The processor-readable medium of claim 13, whereinthe program code for triggering the forced perspective eventconditionally comprises program code for triggering the forcedperspective event only if the user's viewport does not coincide for athreshold duration with an exclusion zone associated with the sphericalframe.
 16. The processor-readable medium of claim 13, wherein theprogram code for triggering the forced perspective event conditionallycomprises program code for triggering the forced perspective event onlyif the user's viewport coincides with an inclusion zone associated withthe spherical frame.
 17. The processor-readable medium of claim 13,further comprising: program code for, where one of a plurality of forcedperspective events is to be triggered at the time of display of thespherical frame, selecting none or one of the plurality of forcedperspective events to be triggered conditionally based at least on theorientation of the user's viewport in relation to the spherical videoframes, the selecting comprising selecting a forced perspective eventonly if the user's viewport coincides with an inclusion zone associatedwith the selected forced perspective event; and program code for, in theevent that one of the plurality of forced perspective events isselected, triggering the selected forced perspective event.
 18. Theprocessor-readable medium of claim 13, further comprising: program codefor, where one of a plurality of forced perspective events is to betriggered at the time of display of the spherical frame, selecting noneor one of the plurality of forced perspective events to be triggeredconditionally based at least on the orientation of the user's viewportin relation to the spherical video frames, the selecting comprisingselecting a forced perspective event only if the user's viewportcoincides for a threshold duration with an inclusion zone associatedwith the selected forced perspective event; and program code for, in theevent that one of the plurality of forced perspective events isselected, triggering the selected forced perspective event.